In my career, I have spent
a lot of time creating text processing algorithms and I've learned a lot of
hard lessons over the years. I want to share with you some general
tips and tricks that will hopefully help you have more tools in your toolbox than I did when I started. The first tip is to plan out your
regular expressions when you can. If you're trying to
extract a known term that only has so many ways of writing it down, you can generate your full list of possible regular
expressions from scratch. For example, if you're looking to
identify patients with hypertension, you may search the term hypertension
or the abbreviation HTN. You may also use stemming, looking for words that
start with "hyperten". This will capture
multiple forms of the word like hypertensive along with hypertension. You can even look for words that
have "hyperten" in the middle of the word to capture things
like antihypertensives, the drugs that are used
to treat hypertension. My second tip is to always look for off-target matches of
your regular expression. With the previous example, if you just search for HTN, and the note has a sentence
"Patient was struck by lightning.", you will find all matches
for the search string. You can get around this by adding a
space before or after the abbreviation. But then you may miss those real matches that just
have punctuation around them. Starting to build
a more complex regular expression to capture each of
these possibilities is vital. Third, as you're reviewing, look for spelling errors and typos. You can either build these into your regular expression or
take an extra step and correct the typos you find using string_replace functions before
applying your regular expression. Fourth, always consider negation and other types of indications
that the match is an accurate. To do so, make sure to review the records that had a match
to your regular expression, go through these cases, and write regular
expressions that exclude matches that had a phrase indicating
that the match was a false positive. For the next couple of tips, I want to share with you
my real-world example of text processing nightmares. I needed to extract warfarin
dose from patient records. Warfarin has a very
complicated dosing schedule where patients may take a different dose
on each day of the week. Now, you might be thinking, "How bad could that be?" Well, let me tell you. There were more than 33,000 unique warfarin doses in
our clinical data warehouse. My general strategy for handling
this problem was to try to make strings that represented the same dose
to have the same format. To do this, you can follow
a couple of general steps. First, remove all leading
and trailing whitespace. This gets rid of the extra spaces on the fringe that don't impact
the meaning of the string. Second, remove words that aren't
necessary to the problem like warfarin, or timing instructions, or drug routes. You can even temporarily replace
the numbers themselves with a placeholder so that the general text format is
combined into a single record. I follow the same approach
when I realized that our providers had managed
to come up with more than 2,400 different ways to type doses that happened on Monday,
Wednesday, and Friday. I created a standard
replacement text that I put in wherever there was a different form
of the date sequence. Third, where you can, try to write combination
regular expressions. For example, this regular
expression matches M, and Mon, and Monday with or
without punctuation afterwards. Finally, a general rule of thumb for
complicated text processing tasks, like my warfarin example, is that your goal should
be to get 80 percent of the data with your text
processing algorithms. Honestly, the last 20
percent is usually so fragmented that you could spend a lot
of time and not make much progress. If you actually do need
the last 20 percent of the data, you may have to resort to manual review. But be sure to confirm that you actually need to review every single record
in that 20 percent. For example, if you have an identification task and a patient already has a note
indicating that they are a case, you do not need to review
their own analyzed notes. As you dig into the course examples
and your own text processing projects, I hope that you find
these tips and tricks helpful. Please consider sharing
your own experiences and tips in the discussion forum.